Image processing apparatus, image processing method, and storage medium

ABSTRACT

An image processing apparatus comprises an image acquisition unit configured to acquire an image, an evaluation value acquisition unit configured to acquire an evaluation value of the image, a reliability acquisition unit configured to acquire reliability of the evaluation value, an evaluation unit configured to evaluate the image based on the evaluation value; and a recording unit configured to add a rating result of the image to the image and record the rating result, wherein the evaluation unit performs rating in accordance with the evaluation value excluding an evaluation value the reliability of which is relatively low.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image processing apparatus thatclassifies an image captured under a poor shooting state and an imagecaptured under a good shooting state.

Description of the Related Art

Conventionally, there is known a camera that classifies an imagecaptured under a poor shooting state which involves camera shake, afocus error, exposure deviation, or the like, and an image capturedunder a good shooting state. For example, Japanese Patent Laid-Open No.2005-184609 discloses a camera that classifies an image based on anauto-focus evaluation value during shooting, a camera shake evaluationvalue acquired by a camera shake amount detection sensor, or the like.

Meanwhile, the auto-focus evaluation value, the camera shake evaluationvalue or the like may not always be acquired stably. For example, in thecase of the auto-focus evaluation value, a correct evaluation valuecannot be acquired for a subject having low contrast. Additionally, in acase where a background is focused on instead of a subject to be focusedon, the auto-focus evaluation value results in a high value due to thebackground being properly focused on.

As described above, it is not necessarily easy to classify an imagecaptured under a poor shooting state and an image captured under a goodshooting state. Then, the conventional technique disclosed in JapanesePatent Laid-Open No. 2005-184609 described above does not take intoaccount the above-described phenomenon, and there is a problem of beingunable to properly evaluate a shooting state depending on a scene to beshot.

SUMMARY OF THE INVENTION

The present invention has been made in view of the problems describedabove, and provides an image processing apparatus that enables properevaluation of a shooting state in a variety of shooting scenes to beshot.

According to a first aspect of the present invention, there is providedan image processing apparatus comprising: at least one processor orcircuit configured to function as: an image acquisition unit configuredto acquire an image; an evaluation value acquisition unit configured toacquire an evaluation value of the image: a reliability acquisition unitconfigured to acquire reliability of the evaluation value; an evaluationunit configured to evaluate the image based on the evaluation value; anda recording unit configured to add a rating result of the image to theimage and record the rating result, wherein the evaluation unit performsrating in accordance with the evaluation value excluding an evaluationvalue the reliability of which is relatively low.

According to a second aspect of the present invention, there is providedan image processing method comprising: acquiring an image: acquiring anevaluation value of the image: acquiring reliability of the evaluationvalue; evaluating the image based on the evaluation value; and adding arating result of the image to the image and recording the rating result,wherein in the evaluation, rating is performed in accordance with theevaluation value excluding an evaluation value the reliability of whichis relatively low.

According to a third aspect of the present invention, there is provideda non-transitory computer-readable storage medium storing a program forcausing a computer to function as respective units of an imageprocessing apparatus, the image processing apparatus comprising: animage acquisition unit configured to acquire an image: an evaluationvalue acquisition unit configured to acquire an evaluation value of theimage; a reliability acquisition unit configured to acquire reliabilityof the evaluation value; an evaluation unit configured to evaluate theimage based on the evaluation value; and a recording unit configured toadd a rating result of the image to the image and record the ratingresult, wherein the evaluation unit performs rating in accordance withthe evaluation value excluding an evaluation value the reliability ofwhich is relatively low.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a configuration of a digitalcamera, according to an embodiment of the present invention.

FIG. 2 is an explanatory view of a configuration of an image capturingelement in an embodiment.

FIG. 3 is a flowchart for explaining an operation of still imageshooting in an embodiment.

FIG. 4 is an explanatory view of a shooting state of still imageshooting in an embodiment.

FIG. 5 illustrates a file structure of image data in an embodiment.

FIG. 6 is an explanatory view of a configuration of image ratinginformation in an embodiment.

FIG. 7 is an explanatory view of a configuration of auto-rating data inan embodiment.

FIG. 8 is an explanatory view of an auto-rating value in an embodiment.

FIG. 9 is a flowchart for explaining an operation of consecutiveshooting of still images in an embodiment.

FIG. 10 is a flowchart for explaining an operation of image capturing byconsecutive shooting in an embodiment.

FIG. 11 is an explanatory view of a configuration of image ratinginformation in an embodiment.

FIG. 12 is an explanatory view of a configuration of a data area in anembodiment.

FIG. 13 is an explanatory view of a configuration of image ratinginformation in an embodiment.

FIG. 14 is an explanatory view of a configuration of a rating data areain an embodiment.

FIG. 15 is an explanatory view of a shooting state of still imageshooting in an embodiment.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, embodiments will be described in detail with reference tothe attached drawings. Note, the following embodiments are not intendedto limit the scope of the claimed invention. Multiple features aredescribed in the embodiments, but limitation is not made an inventionthat requires all such features, and multiple such features may becombined as appropriate. Furthermore, in the attached drawings, the samereference numerals are given to the same or similar configurations, andredundant description thereof is omitted.

FIG. 1 is a block diagram illustrating a functional configuration of adigital camera that is an embodiment of an image processing apparatus ofthe present invention.

In FIG. 1, a control unit 101 is, for example, a CPU. The control unit101 reads out an operation program of each block of a digital camera 100from a ROM 102, and loads and executes the operation program on a RAM103 to control an operation of each block. The ROM 102 is a rewritablenon-volatile memory, and stores a parameter or the like required for anoperation of each block, in addition to the operation program of eachblock of the digital camera 100. The RAM 103 is a rewritable volatilememory, and is used as a temporary storage area of data output during anoperation of each block of the digital camera 100.

An optical system 104 forms a subject image on an image capturing unit105. The image capturing unit 105 includes, for example, an imagecapturing element such as a CCD and a CMOS sensor. The image capturingunit 105 performs photoelectric conversion of an optical image formed bythe optical system 104, and outputs an obtained analog image signal toan A/D conversion unit 106. The A/D conversion unit 106 performs A/Dconversion of an input analog image signal, and stores an obtaineddigital image data in the RAM 103.

An image processing unit 107 applies various image processing such aswhite balance adjustment processing, color interpolation processing,reduction/enlargement processing, and filtering processing to image datastored in the RAM 103.

A recording medium 108 is a removable memory card or the like, andrecords an image stored in the RAM 103 and processed by the imageprocessing unit 107, or an image subjected to A/D conversion by the A/Dconversion unit 106, as a recorded image.

A display unit 109 is a display device such as an LCD. The display unit109 performs through-display of a subject image captured by the imagecapturing unit 105, and also displays various pieces of information. Anoperation unit 110 is an operation member including a release button,and instructs an operation such as auto-focusing and releasing. Notethat the release button is configured to function in a two-step press,and that an image capturing preparation operation starts with ahalf-press of the release button (ON of a switch SW1), and an imagecapturing operation starts with a full press of the release button (ONof a switch SW2).

An evaluation value acquisition unit 111 calculates an evaluation valuesuch as an auto-focus evaluation value from an image data obtained byimage capturing with the image capturing unit 105. A subject detectionunit 112 detects a subject such as a person's face from obtained imagedata.

FIG. 2 is an explanatory view of a configuration of an image capturingelement 105 a provided in the image capturing unit 105. A pixel 202includes a microlens 201 and a pair of photoelectric conversion units203, 204. In the image capturing element 105 a, the pixels 202 areregularly arranged two-dimensionally.

In FIG. 2, it is assumed that an image A and an image B are output as apair of parallax images from the pair of photoelectric conversion units203, 204 regularly arranged two-dimensionally. According to such aconfiguration, a pair of light fluxes passing through different regionsof a pupil of the optical system 104 of FIG. 1 are formed as a pair ofoptical images, and the pair of optical images can be output as a pairof the images A and B. Further, the evaluation value acquisition unit111 of FIG. 1 outputs phase difference distribution of the image A andthe image B as an auto-focus evaluation value (focus information), andalso outputs a determination result for feasibility of focus detection.As for a method for acquiring the phase difference distribution of theimage A and the image B, for example, a method disclosed in JapanesePatent Laid-Open No. 2008-15754 can be used to acquire defocus amountdistribution and simultaneously determine feasibility of focusdetection.

Additionally, the present invention is not limited to this, anddistribution of a shift amount that is an amount of deviation betweenthe image A and the image B may be acquired as the auto-focus evaluationvalue. Additionally, the shift amount may be multiplied by a detectionpitch (arrangement pitch of pixels of the same type) and expressed by aunit of length such as micrometer. Additionally, distribution of a valueobtained by normalizing a defocus amount by a focal depth (2Fδ or 1Fδ: Fis an aperture value and δ is a permissible confusion circle diameter)may be acquired as the auto-focus evaluation value.

FIG. 3 is a flowchart for explaining an operation of still imageshooting in the digital camera 100 of the present embodiment.

First, at S301, an Electronic View Finder image (EVF image) is captured(image acquisition) for an Electronic View Finder (EVF) function todisplay a through image on the display unit 109 of FIG. 1. At S302, aperson's face is detected from the EVF image by using the subjectdetection unit 112 of FIG. 1. As for a face detection method, any knownmethod may be used, and for example, a method disclosed in JapanesePatent Laid-Open No. 2005-286940 can be used to acquire a position and asize of a facial frame and face likeness (likelihood). At S303, therelease button of the operation unit 110 is half-pressed to determinewhether or not the switch SW1 has been turned ON. At S303, S301 and S302are repeated until an auto-focus instruction (switch SW1 turned ON) isissued.

When the auto-focus instruction is issued, Auto Focus (AF) processing isperformed at S304, based on a defocus amount acquired by the evaluationvalue acquisition unit 111 of FIG. 1 by using the EVF image. At S305,the release button of the operation unit 110 is full-pressed todetermine whether or not the switch SW2 has been turned ON. At 305, S301to S304 are repeated until a release instruction (switch SW2 turned ON)is issued.

When the release instruction is issued, a still image for recording(image for recording) is captured at S306. Finally, evaluation of theimage is performed at S307.

In the present embodiment, a person's face is detected from an EVF imagein the face detection at S302, but the present invention is not limitedto this and a person's face may be detected from a still image forrecording. According to such a configuration, a face can be detectedfrom a high-quality still image for recording, and thus detectionprecision can be increased.

In the following, an operation at S307 of FIG. 3 of acquiringreliability (reliability acquisition) and evaluating an image excludingan evaluation value having relatively low reliability will be describedwith reference to FIGS. 4 to 7.

FIG. 4 illustrates an example of a scene to be captured 401, and aperson 402, a building 403, and a tree 404 are aligned from the frontside. An auto-focus frame 406, and a frame group 405 of defocus amountdistribution that can be acquired by the evaluation value acquisitionunit 111 of FIG. 1 are indicated. The auto-focus frame 406 and defocusdistribution by 3×3 frames surrounding the auto-focus frame 406 can beacquired. In the present embodiment, defocus amount distribution(evaluation value acquisition) of only a portion of a screen near anauto-focus frame is acquired, but the present invention is not limitedto this, and is also applicable in a case where defocus amountdistribution has been acquired across a whole screen.

A still image 407 in a case where a person's face is successfullyfocused on is indicated. Still images 408 and 409 in a case where abackground has been erroneously focused on are indicated. In the stillimage 408, the tree 404 is focused on, and in the still image 409, thebuilding 403 is focused on. In AF processing, a position of anauto-focus frame is determined also with reference to a result of facedetection, and the auto-focus frame position is determined also withreference to other indicator such as subject contrast. Therefore, when ahigh-contrast subject is present in a background, an auto-focus framemay be moved toward the background and, as a result, the background maybe focused on erroneously as in the still images 408 and 409.

Additionally, there is only one person's face to be detected in theexample of FIG. 4, but in a case where a plurality of persons' faceshave been detected, a subject to be focused on is selected, based on asubject position, a subject likelihood, or the like output by thesubject detection unit 112 of FIG. 1. For example, a method forprioritizing a subject having a higher subject likelihood, prioritizinga subject at the center of a screen, or prioritizing a subject closer toan auto-focus frame position or a position specified by a user with atouch panel provided on the display unit 109 of FIG. 1 is conceivable.Alternatively, the subject detection unit 112 of FIG. 1 may also performline-of-sight detection in addition to face detection, and a subjecthaving a line of sight oriented in the direction of the digital cameramay be prioritized. Further, a frame may be displayed in a selectedsubject during still image playback, and thus a user can recognize whichsubject has been selected.

In the present embodiment, a face is detected from a whole screen, butthe present invention is not limited to this, and detection may belimited to a face present in a portion of a screen. For example,detection may be limited to the defocus amount distribution acquisitionrange indicated by 405 of FIG. 4 and a periphery of this range.According to such a configuration, a face detected from a region havinglow importance for image evaluation can be excluded, and thus a risk oferroneous face detection can be reduced.

As for a configuration of a still image file, a known file configurationsuch as the Exif standard may be used, and for example, an optimal imagedata file structure for group playback compliant with the Exif standardis disclosed in Japanese Patent Laid-Open No. 2012-165080.

FIG. 5 is an explanatory view of a configuration of a still image fileto be recorded in the recording medium 108 of FIG. 1. Image data 1501constituting a still image file includes a marker SOI (1502) at thehead, and the marker indicates the start of an image, and is followed byan application marker APP1 (1503). The application marker APP (1503)includes a size (1504) of the APP1, an identification code (1505) of theAPP1, DateTime (1506) indicating the creation date and time of theimage, DateTimeOriginal (1507) indicating the date and time when theimage data is generated, RatingInformation (1508) indicating imagerating information, miscellaneous shooting information (1509), and athumbnail image (1510).

Additionally, as illustrated in FIG. 5, the image data to be recordedincludes a quantization table DQT (1511), a Huffman table DHT (1512), astart-of-frame marker SOF (1513), a start-of-scan marker SOS (1514), andcompression data 1515. The image data is then terminated with a markerEOI (1516) indicating the end of the image data.

The file structure illustrated in FIG. 5 is a structure definedaccording to the Exif standard, and is recognizable as an Exif structureby referring to the APP1 (1503) code and the identification code (1505).

In the present embodiment, the file structure is compliant with the Exifstandard, but the present invention is not limited to this, and the filestructure may be compliant with other standards, or separately, a filestructure suitable for realizing the present invention may be definednewly and used.

FIG. 6 is an explanatory view of a configuration of image ratinginformation to be recorded in a header (application marker APP1) of astill image file to be recorded. The image rating information includes avalue of user rating, a type of auto-rating, a subject of auto-rating,and data of auto-rating.

The user rating is an image evaluation value manually evaluated(manually set) by a user. For example, a value 0 indicates that there isno rating result, and values 1 to 5 indicate that there is a ratingresult with a higher value indicating a higher rating result. The userrating is recorded separately from image evaluation automaticallyperformed by the digital camera. Accordingly, when narrowing search isperformed on an image, separate (individual) search can be performed,and thus convenience is improved.

The type of auto-rating is a type of evaluation value for evaluation inthe image evaluation at S307 of FIG. 3. For example, a numeral 1indicates a focus evaluation value, and when a numeral 0 is set, absenceof auto-rating data is indicated. Additionally, it is assumed thatnumerals 2 to 9 indicate reserved areas for future expansion. In thepresent embodiment, focus is evaluated as image evaluation, but thepresent invention is not limited to this. Exposure (exposureinformation), contrast (contrast information), shake (shakeinformation), or the like may be evaluated, and the rating result may beassigned to any of the reserved areas 2 to 9.

The subject of auto-rating is a type of subject detected by the subjectdetection unit 112 of FIG. 1. For example, a numeral 1 indicates aperson and numerals 2 to 9 indicate reserved areas. Alternatively, asubject indicated by an auto-focus frame may be specified without usinga result provided by the subject detection unit 112, and in that case, 0is set. In the present embodiment, the subject detection unit 112detects a person's face and thus 1 is set. However, the presentinvention is not limited to this, and a subject other than a person suchas an auto-focus frame assigned to 0, an animal, a vehicle, a subjectspecified by a user with a touch panel on the display unit 109 of FIG. 1may be detected and assigned to any of the reserved areas 2 to 9.

FIG. 7 is an explanatory view of a configuration of auto-rating data.The auto-rating data includes a value of auto-rating, auto-ratingattributes 1 to 3, a position and a size of a subject frame, and alikelihood of a subject frame.

The value of auto-rating is an image evaluation value of automaticevaluation by the digital camera. For example, a value 0 indicates thatthere is no rating result, and values 1 to 5 indicate that there is arating result with a higher value indicating a higher rating result.

The auto-rating attribute 1 is set to 0 when the subject detection unit112 of FIG. 1 has successfully detected a subject. The auto-ratingattribute 1 is set to 1 when the subject detection unit 112 cannotdetect a subject. Feasibility of subject detection is determined by alikelihood output by the subject detection unit 112. The auto-ratingattribute 2 is set to 0 when it is determined that focus detection bythe evaluation value acquisition unit 111 of FIG. 1 is feasible, and theauto-rating attribute 2 is set to 1 when it is determined that the focusdetection is unfeasible. The auto-rating attribute 3 is set to 0 when anevaluation value has been output at a position where a subject has beendetected, and the auto-rating attribute 3 is set to 1 when no evaluationvalue has been output. For example, in 407 and 408 of FIG. 4, a defocusamount is acquired at a position of a face that is a subject, and thus 0is set. On the other hand, in 409, no defocus amount has been acquiredat a position of a face which is a subject, and thus 1 is set.

According to such a configuration, a user can check afterwards thereason why a still image having no image rating result has not beenprovided with a rating result, and thus convenience in determining asearch condition for performing narrowing search of a still image usingan image evaluation result can be improved.

The position and the size of a subject frame, and the likelihood of asubject frame are calculated based on a result output from the subjectdetection unit 112 of FIG. 1.

FIG. 8 is an explanatory view 701 of a relation among the rating value,the subject detection result, and the defocus amount of FIG. 7. In thecase where subject detection is unfeasible, the rating value is set to0. Even when subject detection is feasible, the rating value is set to 0in a case where position deviation between a subject and an auto-focus(AF) frame is significant exceeding TH_POS. For example, in 409 of FIG.4, there is significant deviation between a position of a face which isa subject and an AF frame position and thus the rating value is set to0. Even when position deviation between a subject and an AF frame is assmall as TH_POS or less, the rating value is set to 0 in a case wherefocus detection is unfeasible. In the case where focus detection isfeasible, the smaller the defocus amount at a detected subject position,the higher the rating value is set (a defocus amount threshold isdetermined by a unit of 1Fδ: F is an aperture value and δ is apermissible confusion circle diameter).

When the rating value is set only based on a defocus amount of an AFframe, a high rating value is set in a case where a background has beenerroneously focused on such in 408 or 409 of FIG. 4. However, when therating value is set based on a defocus amount at a detected subjectposition as in the present embodiment, a correct rating value can beset.

In the present embodiment, the rating value is set to 0 in a case wherea position of a subject and a position of an AF frame deviatesignificantly exceeding TH_POS, but the present invention is not limitedto this. For example, instead of the defocus amount, an amount ofposition deviation between a subject and an AF frame may be used as anevaluation value to determine the rating value as in 702 of FIG. 8. Thatis, in 409 in the case of FIG. 4, since auto-focus is performed at aposition far away from a position where a face which is a subject hasbeen detected, a good focus state is not expected, and thus a lowevaluation value (1) is set in the rating value.

According to such a configuration, even when no defocus amount is outputat a position where a subject is detected as in 409 of FIG. 4, an imagecan be evaluated by using an amount of position deviation between asubject and an AF frame as an evaluation value.

FIG. 9 is a flowchart for explaining an operation of consecutiveshooting of still images. S301 to S305 of FIG. 9 are similar to those ofthe flowchart of still image shooting of FIG. 3. In FIG. 9, when arelease instruction is issued (switch SW2 turned ON) at S305, stillimages for recording are captured by consecutive shooting at S806.

FIG. 10 is a flowchart for explaining an operation of image capturing byconsecutive shooting at S806 of FIG. 9.

First, at S901, a still image for recording is captured, and at S902,image evaluation is performed by using results of face detection and AFprocessing performed before still image capturing.

Next, EVF image capturing, face detection, and AF processing areperformed at S903. S904, and S905, respectively, for subsequent stillimage capturing. At S906, a release instruction is performed (switch SW2turned ON), and S901 to S905 are repeated while the release instructionis continued, and image capturing ends when the release instruction isreleased (NO at S906).

In the present embodiment, image evaluation is performed by using theresults of face detection and AF processing by EVF image capturingperformed before still image capturing, but the present invention is notlimited to this. Results of face detection and AF processing bytemporally later EVF image capturing and performed for subsequent stillimage capturing may be used. Accordingly, even in a case where no facehas been detected in an EVF captured image captured before still imagecapturing, it is possible to add an image evaluation result indicatingthat there is a rating result for the still image to be recorded,provided that a face is detected in an EVF captured image captured afterstill image capturing.

Additionally, in a case where no face has been detected in both an EVFcaptured image captured before still image capturing and an EVF capturedimage captured after still image capturing, face detection may beperformed in still image capturing. In this case, results of both the AFprocessing performed before still image capturing and the AF processingperformed after still image capturing may be composited to estimatedefocus amount distribution of captured still images.

In the present embodiment, the AF processing is performed by using EVFimage capturing prior to still image capturing, but the presentinvention is not limited to this. AF processing may be performed byusing still image capturing of one frame before the current still imagecapturing, without performing the EVF image capturing at S903 of FIG.10. In this case, image evaluation is performed by using results of facedetection and AF processing performed in the still image capturing ofthe same frame as the still image to be recorded. According to such aconfiguration, a consecutive shooting frame rate can be increased asmuch as EVF image capturing is eliminated, and defocus amountdistribution acquired in the same frame as the still image to berecorded can be used. Thus, accurate image evaluation results are addedto the recorded still image without a time lag.

In the present embodiment, the subject detection unit 112 of FIG. 1detects a face, but the present invention is not limited to this, andbehavior of a subject may be detected from a time series image groupobtained by EVF image capturing or still image capturing. For example, aplayer spiking in a volleyball game may be detected as a subject, or aplayer dribbling in a soccer game may be detected as a subject. Inaddition to the behavior detection, a front-running subject in a hurdlerace may be detected (state detection) or a frequently appearing player(with a high appearance frequency) may be detected. According to such aconfiguration, even in the case of the presence of a plurality ofsubjects, a desired subject can be detected efficiently, and imageevaluation for the desired subject can be performed.

In the present embodiment, only one subject type that is a face can bedetected at a time by the subject detection unit 112 of FIG. 1, but thepresent invention is not limited to this, and a plurality of subjecttypes may be detected simultaneously.

FIG. 11 is an explanatory view of a configuration of image ratinginformation to be recorded in a header of a still image file in the caseof simultaneously detecting a plurality of subject types. The imagerating information includes a value of user rating, a type ofauto-rating, the number of subject classes, pointers to data areas 1 and2, and data areas 1 and 2.

The value of user rating, and the type of auto-rating are similar tothose of FIG. 6. The number of subject classes is the number of subjecttypes to be detected simultaneously. The data areas 1 and 2 are areasstoring image rating information for each subject type, and the pointersto the data areas 1 and 2 are offset addresses to respective areas.

FIG. 12 is an explanatory view of a configuration of the data areas ofFIG. 11. A data area includes a type of subject class, the number ofsubjects, and auto-rating data of subjects 1 and 2.

The type of subject class is similar to the subject of the auto-ratingof FIG. 6. The number of subjects is the number of subjects to beregistered with a target data area. For example, in a case where thetype of subject class of the target data area is a person, when twopersons are detected, a value 2 is set. The auto-rating data of thesubjects 1 and 2 is similar to the auto-rating data of FIG. 6.

According to such a configuration, even when a plurality of subjecttypes such as a person and an animal are appearing in the same stillimage, a desired subject can be extracted without omission whennarrowing search or the like of a still image using image evaluationresults. Additionally, in a case where a plurality of persons exist withthe same type of subject person being set, image evaluation results canbe recorded separately, and thus in performing narrowing search of astill image, it is possible to select, for example, use of an imageevaluation result of a person at the center of an image, or use of animage evaluation result of a person close to an AF frame even when avalue of a subject likelihood is low.

In the present embodiment, only one type of evaluation value that is thedefocus amount can be acquired at a time by the evaluation valueacquisition unit 111 of FIG. 1, but the present invention is not limitedto this, and a plurality of evaluation values may be acquiredsimultaneously. In addition to the defocus amount, an evaluation valuesuch as exposure, contrast, and shake may be acquired simultaneously.

FIG. 13 is an explanatory view of a configuration of image ratinginformation to be recorded in a header of a still image file in the caseof acquiring a plurality of evaluation values simultaneously. The imagerating information includes a value of user rating, the number of typesof auto-rating, pointers to rating data areas 1 and 2, and rating dataareas 1 and 2.

The value of user rating is similar to that of FIG. 6. The number oftypes of auto-rating is the number of types of evaluation values to beacquired simultaneously. The rating data areas 1 and 2 are areas storingimage rating information for each type of evaluation value, and thepointers to the rating data areas 1 and 2 are offset addresses torespective areas.

FIG. 14 is an explanatory view of a configuration of the rating dataarea of FIG. 13. The rating data area includes a type of auto-rating,the number of subject classes, pointers to data areas 1 and 2, and dataareas 1 and 2.

The type of auto-rating is a type of evaluation value to be registeredin a target auto-rating data area. For example, it is assumed that anumeral 1 indicates a focus evaluation value, 2 indicates an exposureevaluation value, 3 indicates a contrast evaluation value, 4 indicates ashake evaluation value, and when a numeral 0 is set, it indicatesabsence of target auto-rating data area. Additionally, it is assumedthat numerals 5 to 9 indicate reserved areas for future expansion. Thenumber of subject classes, the pointers to the data areas 1 and 2, andthe data areas 1 and 2 are similar to those of FIG. 11.

According to such a configuration, narrowing search of a still imageusing image evaluation results can be performed from variousperspectives in addition to focus, and thus user convenience can beimproved.

In the present embodiment, image evaluation is performed based on anevaluation value of a subject of interest, but the present invention isnot limited to this, and framing of a whole screen may be analyzed toperform image evaluation.

FIG. 15 is an explanatory view of an operation of analyzing the framingof a whole screen to perform image evaluation. FIG. 15 illustrates anexample of a scene to be captured 1401, and 402 to 404 are similar tothose of FIG. 4. A person 1402 is a main subject, and is located morefrontward than a person 402. An auto-focus frame 1404 is indicated, anda frame group 1403 of defocus amount distribution can be acquired by theevaluation value acquisition unit 111 of FIG. 1. Defocus distributionacross a whole screen is acquired.

For example, the framing of a whole screen may be analyzed to provide animage with a high rating result in a case where only a person that is amain subject is focused on, and other persons are not focused on. In anexample of such a still image 1405, the person 1402 that is a mainsubject is focused on, and the other person 402 is not focused on.According to such a configuration, a high rating result is provided to astill image in which an accidentally captured obstructive subject (suchas a passer-by walking behind) is kept inconspicuous, and only a mainsubject is impressively outstanding. This is useful in extracting animpressive portrait scene with narrowing search of a still image.

Additionally, for example, the framing of a whole screen may be analyzedto provide an image with a high rating result in a case where there area large number of persons focused on. In an example of such a stillimage 1406, there are two persons of the person 1402 that is a mainsubject and the person 402, and both the two persons are focused on. Onthe other hand, in the still image 1405, only one of the two persons isfocused on, and thus the still image 1405 is provided with a lower imagerating result than the still image 1406. Such a configuration is usefulin extracting, by narrowing search of a still image, a scene having asatisfactory image quality as a group photograph.

In the present embodiment, narrowing search of a still image isperformed in performing playback by using results of image evaluation,but the present invention is not limited to this. In a case whereresults of image evaluation are low, a target still image may not berecorded, or still image capturing itself may be stopped. According tosuch a configuration, recording capacity of the recording medium 108 ofFIG. 1 can be saved.

In the present embodiment, narrowing search of a still image usingresults of image evaluation is performed to playback the still image,but the present invention is not limited to this, and a playback methodof a narrowed-down still image may be changed. For example, a moredetailed focus checking can be performed by using defocus amountdistribution acquired by the evaluation value acquisition unit 111 ofFIG. 1 to strengthen coloration or sharpness in a region focused on in astill image or, conversely, strengthen coloration or blur in a regionnot focused on in a still image, or by reproducing defocus amountdistribution instead of a still image.

In the present embodiment, image evaluation is performed in a digitalcamera, but the present invention is not limited to this, and imageevaluation may be performed in an external apparatus such as a mobiledevice, a personal computer, or a cloud server device. For example, whenimage evaluation is performed in an external device, subject detectioncan be performed by the external device, and not only simple subjectdetection of a person or an animal, but also advanced subject detectionsuch as determination of, for example, even a specific breed of dog canbe performed. Therefore, it is possible to realize image narrowingsearch more suited for application of a user.

In the present embodiment, the defocus amount distribution acquired inthe AF processing at the time of shooting is used for image evaluation,but the present invention is not limited to this, and the defocus amountdistribution may be acquired after shooting.

For example, acquisition of a phase difference image between the image Aand the image B is performed at the time of shooting, and acquisition ofthe defocus amount distribution is performed after shooting, and thus,the defocus amount distribution can be acquired at any position withoutbeing limited to an area near an AF frame. Therefore, it is possible toreduce the number of still images to be evaluated to have no imagerating result, and improve the convenience of narrowing search of astill image.

Additionally, when rating based on an evaluation value is high, an imagerendered based on the evaluation value or an evaluation value mapindicating spatial distribution of the evaluation value may be displayedon the display unit 109.

In the present embodiment, the defocus amount distribution is acquiredby an on-imaging surface phase difference method, but the presentinvention is not limited to this. The defocus amount distribution may beacquired by other focus detection method such as a contrast auto-focusmethod or a DFD (Depth From Defocus) method.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2019-160662, filed Sep. 3, 2019, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising: atleast one processor or circuit configured to function as: an imageacquisition unit configured to acquire an image; an evaluation valueacquisition unit configured to acquire an evaluation value of the image;a reliability acquisition unit configured to acquire reliability of theevaluation value; an evaluation unit configured to evaluate the imagebased on the evaluation value; and a recording unit configured to add arating result of the image to the image and record the rating result,wherein the evaluation unit performs rating in accordance with theevaluation value excluding an evaluation value the reliability of whichis relatively low.
 2. The image processing apparatus according to claim1, wherein the at least one processor or circuit is configured tofurther function as a subject detection unit configured to detect asubject from the image, wherein the evaluation unit determines thatreliability of the evaluation value is relatively high at a subjectposition detected by the subject detection unit.
 3. The image processingapparatus according to claim 2, wherein the subject detection unitfurther detects a likelihood of a subject, and the evaluation unit, in acase where the subject detection unit has not detected a subject thelikelihood of which is relatively high, does not perform evaluation ofan image reliability of which is determined to be relatively low.
 4. Theimage processing apparatus according to claim 2, wherein the subjectdetection unit detects at least a first subject and a second subject,and the evaluation unit performs evaluation of the first subject andevaluation of the second subject individually.
 5. The image processingapparatus according to claim 3, wherein the at least one processor orcircuit is configured to further function as a setting unit configuredto set a subject to be prioritized the reliability of which isdetermined to be relatively high, based on the subject position or thelikelihood.
 6. The image processing apparatus according to claim 5,wherein the setting unit prioritizes a subject located near anauto-focus frame or a position indicated by a user.
 7. The imageprocessing apparatus according to claim 2, wherein the at least oneprocessor or circuit is configured to further function as a setting unitconfigured to set a subject to be prioritized reliability of anevaluation value of which is determined to be relatively high, based onappearance frequency of a subject, action detection of a subject, orstate detection of a subject.
 8. The image processing apparatusaccording to claim 5, wherein the evaluation value acquisition unitacquires, as a new evaluation value, a difference between an evaluationvalue of a prioritized subject and an evaluation value of other subject,and the evaluation unit composes the evaluation value of the prioritizedsubject and the evaluation value of the other subject to evaluate animage.
 9. The image processing apparatus according to claim 2, wherein,in a case where the subject detection unit has detected a plurality ofsubjects, the evaluation unit evaluates an image based on a number ofsubjects having a relatively high evaluation value.
 10. The imageprocessing apparatus according to claim 2, wherein the evaluation unitdetermines the reliability based on position deviation between a targetsubject of auto-focus and a detected subject.
 11. The image processingapparatus according to claim 1, further comprising a display deviceconfigured to display an image, wherein an image searched for based onthe evaluation value is displayed on the display device.
 12. The imageprocessing apparatus according to claim 11, wherein, in a case where arating result based on the evaluation value is relatively high, an imagerendered based on the evaluation value or an evaluation value maprepresenting spatial distribution of the evaluation value is displayedon the display device.
 13. The image processing apparatus according toclaim 2, wherein an image having a relatively low rating result based onthe evaluation value is not recorded in the recording unit.
 14. Theimage processing apparatus according to claim 2, wherein the imageacquisition unit acquires an electronic view finder image and an imagefor recording, and the subject detection unit detects a subject from theelectronic view finder image.
 15. The image processing apparatusaccording to claim 2, wherein the image acquisition unit acquires anelectronic view finder image and an image for recording, and the subjectdetection unit detects a subject from the image for recording.
 16. Theimage processing apparatus according to claim 1, wherein the evaluationvalue includes any of image focus information, exposure information,contrast information, and shake information.
 17. The image processingapparatus according to claim 16, wherein the focus information includesinformation based on a shift amount representing parallax or informationbased on a defocus amount.
 18. The image processing apparatus accordingto claim 1, further comprising a manual setting device by which a usermanually sets a rating result.
 19. An image processing methodcomprising: acquiring an image; acquiring an evaluation value of theimage; acquiring reliability of the evaluation value; evaluating theimage based on the evaluation value; and adding a rating result of theimage to the image and recording the rating result, wherein in theevaluation, rating is performed in accordance with the evaluation valueexcluding an evaluation value the reliability of which is relativelylow.
 20. A non-transitory computer-readable storage medium storing aprogram for causing a computer to function as respective units of animage processing apparatus, the image processing apparatus comprising:an image acquisition unit configured to acquire an image; an evaluationvalue acquisition unit configured to acquire an evaluation value of theimage; a reliability acquisition unit configured to acquire reliabilityof the evaluation value; an evaluation unit configured to evaluate theimage based on the evaluation value; and a recording unit configured toadd a rating result of the image to the image and record the ratingresult, wherein the evaluation unit performs rating in accordance withthe evaluation value excluding an evaluation value the reliability ofwhich is relatively low.